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ABSTRACT 



This packet introduces some concepts in evaluation and 
accountability for mental health professionals in schools. The first section 
presents two briefs from the Clearinghouse of the Center for Mental Health in 
Schools. The first, "Mental Health in Schools: Quality Control, Evaluating 
Outcomes, and Getting Credit for All You Do,” highlights the importance of 
planning evaluation to measure outcomes and enhance the quality of 
intervention and student benefits. The second brief, "Evaluation, 
Accountability, and Mental Health in Schools," focuses on accountability 
indicators and how to gather data related to such indicators. The second 
section of the packet provides "A Quick Overview of Some Basic Resources, " 
including 22 selected references, a list of 5 guidebooks and models, a list 
of 8 agencies and Web sites for technical assistance, and a list of experts 
from the Center's consultation cadre. An evaluation flow chart and some 
accompanying material from a guide created by the National Institute on Drug 
Abuse form the third section. The final section contains two reports from the 
Center for School Mental Health Assistance, "Quality Assurance" and 
"Documenting Effectiveness of School Mental Health Programs."' (SLD) 
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From the Center's Clearinghouse ... 




An introductory packet on 



Evaluation and Accountability: 
Getting Credit for All You Do! 



U S- DE pART M E N T OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
S_- CENTER (ERIC) 

This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 

• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
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P e rry tueison 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




The Center is co-directed by Howard Adelman and Linda Taylor and operates under the auspices of the v „ 
School Mental Health Project, Dept, of Psychology, UCLA, Los Angeles, CA 90095-1563 -- f 

Phone:(310)825-3634. r 

U.S. Depot tmem of HedA & Hum 

Support comes in part from the Department of Health and Human Services, 

Public Health Service, Health Resources and Services Administration, 

Maternal and Child Health Bureau, Office of Adolescent Health. 
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UCLA CENTER FOR MENTAL HEAL TH IN SCHOOL S' 



Center f 



Under the auspices of the School Mental Health Project in the Department of 
Psychology at UCLA, our center approaches mental health and psychosocial 
concerns from the broad perspective of addressing barriers to learning and 
promoting healthy development. Specific attention is given policies and 
strategies that can counter fragmentation and enhance collaboration between 
school and community programs. 



MISSION: To improve outcomes for young people 

by enhancing policies, programs, and 
practices relevant to mental health 
in schools. 



Through collaboration, the center will 

■ enhance practitioner roles, functions and competence 

■ interface with systemic reform movements to 

strengthen mental health in schools 

■ assist localities in building and maintaining their 

own infrastructure for training, support, and 
continuing education that fosters integration of 
mental health in schools 



Consultation Cadre Clearinghouse 

Newsletter National & Regional Meetings 

Electronic Networking 
Guidebooks Policy Analyses 



Co-directors: Howard Adelman and Linda Taylor 

Address: UCLA, Dept, of Psychology, 405 Hilgard Ave., Los Angeles, CA 90095-1563. 

Phone: (310) 825-3634 FAX: (310)206-8716 E-mail: smhp@ucla.edu 

Website: http://smhp.psych.ucla.edu/ 

* In 1 996, two national training and technical assistance centers focused on mental health in schools 
were established with partial support from the U.S. Department of Health and Human Services, 

Public Health Service, Health Resources and Services Administration, Maternal and Child Health 
Bureau, Office of Adolescent Health. As indicated, one center is located at UCLA; the other is at 
the University of Maryland at Baltimore and can be contacted toll free at 1 -(888) 706-0980. 
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What is the Center’s Clearinghouse? 



The scope of the Center’s Clearinghouse reflects the School Mental He^thProj_ _ 
mission — to enhance the ability of schools and their surrounding promote 

address mental health and psychosocial barriers to student le ajn ,n g ^ P d 
healthy development. Those of you working so hard to address these concerns need 
ready access to resource materials. The Center's* Clearinghouse is y r 
specialized resources, materials, and information. The staff supplemen , c P • 
and disseminates resources on topics fundamental to our mission. As w 
what is available across the country, we are building systems to connec y°u 
wide variety of resources. Whether your focus is on an individua , y> 

classroom, a school, or a school system, we intend to be of service y • 
evolving catalogue is available on request; eventually it will 
electronically over the Internet. 



What kinds of resources, materials, and information are available. 

We can provide or direct you to a variety of resources, materials, and information that we 
have categorized under three areas of concern: 



• Specific psychosocial problems 

• Programs and processes 

• System and policy concerns 



Among the various ways we package resources are our Introductory Packets, ^s°“'’ce-4id 
pSSk special reports, guidebook and ccmMng 

overview discussions of major topics, descriptions of model programs, references 
publications, access information to other relevant centers, organizations, VO( : ac - , P ' 
and Internet links, and specific tools that can guide and asstst with training a«ivUy 
student/family interventions (such as outlines, checklists, ujtntments, Md o 
that can be copied and used as information handouts and aids or p )• 



Accessing the Clearinghouse 



• E-mail us at 

• FAX us at 

• Phone 
. Write 



smhp@ucla.edu 
(310) 206-8716 

School MenSlHealth Project/Center for Mental Health in Schools, 
Dept, of Psychology, Los Angeles, CA 90095-1 56 j 



Check out recent additions to the Clearinghouse on h °^^^ psych . ucIa . edu 



All materials from the Center’s Clearinghouse are available for a minimal fee to cover fte cost of 
foo^nr^dUng, and postage. Eventtiklly, we plan to have some of this material and other 
Clearinghouse documents available, at no-cost, on-line for those with Internet access. 



If you know of something we should have in the clearinghouse, let us know. 
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Evaluation and Accountability 



Systematic evaluation is increasingly sought to guide operations, 
to assure legislators and planners that they are proceeding on 
sound lines, and to make services responsive to their publics. 

Lee Cronbach and colleagues 



This introductory packet contains: 

I. Two Clearinghouse Briefs from our Center: 

• Mental Health in Schools: Quality Control, Evaluating Outcomes, and Getting 
Credit for All You Do. 

• Evaluation, Accountability, and Mental Health in Schools 

II. A Quick Overview of Some Basic Resources 

• Selected References 

• Guidebooks and Models 

• Agencies and Websites for Technical Assistance 

• Consultation Cadre 

III. An Evaluation Flow Chart and some accompanying material from the guide by the 
National Institute of Drug Abuse 

IV. Two reports from Center for School Mental Health Assistance based on 
Critical Issue Planning Sessions 

• I: Quality Assurance 

• II: Documenting Effectiveness of School Mental Health Programs 



^Hhjg 



A BRIEF DISCUSSION OF: 




MENTAL HEALTH IN SCHOOLS: 

QUALITY CONTROL, EVALUATING OUTCOMES, AND 
GETTING CREDIT FOR ALL YOU DO 

We approach mental health activity in schools as one facet of a comprehensive, 
integrated approach to addressing barriers to learning and enhancing healthy 
evelopment. The intent of all such activity, of course, is to enhance outcomes 
f , . , , 1 ° r children and adolescents. However, enhancing outcomes for the large number 

ot those in need of help usually involves addressing the systems that determine such outcomes (e g 
families, education support programs, school-based health centers, off-site services, the community at 
arg . e '‘. Moreover, it is important to proceed with a holistic perspective (e.g., viewing children in the 
context of families and communities). Such a perspective fosters appreciation of relationships among 
individuals specific aspects of systems, and the system as a whole. Given this comprehensive orientation 
served” 13 ^ 11 1S evident that eva *uation involves more than measuring outcomes for individuals 






Broadly stated, evaluation should be planned and implemented in ways that measure outcomes and much 
more with a view to enhancing the quality of intervention efforts and the long-term benefits for students 
and society. The following sections highlight a few ideas along these lines. 

Evaluation that Fosters Quality Improvement 

One purpose of outcome evaluation is to provide feedback on efficacy so processes can be revised and fine- 
tuned. Such formative evaluation also includes information on participants, approaches, resources 
implementation strategies, program organization, staffing, operational policies and practices. It also should 
; C ™ f on *5 characteristics of the system's "clients" - who they are, what they want and need, how 
y diner from those in other locales - as a prerequisite for effective planning and as another basis for 
interpreting the appropriateness of observed processes and outcomes. (That is, it is essential to understand 
the status of clients before an intervention is implemented, not only to be aware of their needs but 
ultimately to make appropriate judgments about intervention outcome efficacy.) 

Thus, formative evaluation includes data gathering and analyses focused on such matters as 



• needs and assets, goals and desired outcomes, resources, and activities 

challenges and barriers to mental health intervention and the integration of such interventions with other 
activity designed to address barriers to learning, as well as with the instructional and management 
components of schools and communities 

characteristics of families and children in each locale, with special focus on targeted groups 

• initial outcomes. 



Formative evaluation data may be gathered on and from samples of all parties who have a stake in the 
intervention (e.g., school staff, students and their families, other stakeholders, community agencies and 
so forth). The information is used to judge the "fit" of prerequisite conditions and processes. Methods 
used include review ofdocuments and records, checklists, surveys, semi-structured interviews, focus group 
discussions, observations, and direct assessment of clientele. A well-designed information management 
system can be a major aid (e.g., providing data on identified needs and current status of individuals and 
resources). In this respect, an advanced technology can play a major role (e.g., a computerized system that 
is properly designed can provide access to information in other computer-based data systems containing 
relevant information on clients and processes). 
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To be maximally useful, a data set should allow for baseline and subgroup comparisons and include 
multiple variables so that findings can be disaggregated during analysis Of particular interest are data 
differentiating clients in terms of demographics, initial levels of motivation and development, and type, 
severity, and pervasiveness of problems. With respect to process, it is useful to have data differentiating 
stages of program development and differences in program quality. 

Optimally, the date gathered should allow for formative-leading-to-summative evaluations. Designing s 
formative evaluation system that over time yields summative findings facilitates ongoing planning in ways 
that improve processes and thus outcomes. At the same time, such an approach builds a system for 
validating interventions. 



Evaluation Focused on Results 

To begin with, it will help to clarify our definition of some terms that are used throughout this section. 
Aims are extremely abstract statements of intended outcomes that encompass many goals and objectives; 
this usually means an aim can only be accomplished over an extensive time period (e.g., many years). 
Goals are somewhat less abstract statements encompassing many objectives; thus, a goal usually requires 
a somewhat extended period of time to accomplish. Objectives are meant to be less abstract and more 
immediately accomplishable than the goal that encompasses them. A standard is defined as a statement 
about what is valued. Standards are used to (a) judge and promote quality, (b) clarify goals, and (c) 
promote change. In evaluating efficacy, standards are operationalized in terms of specific criteria upon 
which judgments of immediate and potential long-term efficacy can be made. Indicators of efficacy are 
measurable variables that can be accessed from various sources through use of specific data gathering 
strategies and tools. 



As emphasized above, while the intent of mental health activity in schools is to enhance outcomes for 
students, the effort must also address the systems that determine such outcomes. Thus, the following 
discussion outlines intended impact not only on students, but on families and community, and on programs 
and systems. 

Student Outcomes 

Efforts to address mental health concerns and other barriers to learning include enhancing receptivity to 
instruction through facilitating positive academic, social, emotional, and physical development. In this 
section, we focus first on outcomes related to facilitating such development; then, the emphasis shifts to 
prevention and correction of emotional, behavioral, learning, and health problems. 

(1) Outcomes reflecting enhanced receptivity to instruction. Teaching and learning are 
transactional. Students (and teachers) bring certain capacities and attitudes (abilities, expectations, values) 
accumulated and established over time. These provide the foundation upon which teaching tries to build. 
Students also come with current physiological and psychological states of being that can facilitate or inhibit 
learning at any given time. Efforts to enhance receptivity to instruction focus on ensuring there is a good 
instructional match with the student's capacities, attitudes and current state of being. While this is 
especially necessary for those manifesting serious problems, it is a fundamental concern related to all 
learners. 

The aim of enhancing receptivity to instruction involves ensuring that students have the opportunity to 
acquire the types of basic abilities, expectations, and values that enable learning. The aim also 
encompasses the need for schools to respond appropriately to variations in students' current states of being 
(e.g., ensuring the opportunity to learn by providing breakfast and lunch programs to combat hunger, 
responding to personal problems and crises with support and guidance). 



As is highlighted by the goals and objectives outlined in Exhibit A, the ultimate aim is to ensure that 
students develop effective levels of functionality - academically, socially, emotionally, and physically. 
(With respect to social-emotional functioning, aims are sometimes referred to as personal qualities, 
interpersonal functioning, the affective domain, and so forth. Physical functioning often is discussed as 
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physical and health education.) From a developmental perspective, the aim encompasses concerns for 
ensuring a "healthy start," a safe school envirorunent, preparation (readiness) for school, facilitating 
continued positive development in all areas, facilitating progress with respect to developmental tasks at 
each stage of development, enhancing areas of personal interest and strength, and fostering a psychological 
sense of community. As with all curricular goals, desired outcomes in these areas reflect (a) intended uses 
(communication, reasoning, problem solving, making relationships and connections, and creativity) 
and (b) factors related to intrinsic motivation (personal valuing and expectations of efficacy - including 
confidence in one's abilities). 

The goals and objectives outlined in Exhibit A provide a frame of reference for designing programmatic 
activity to facilitate development related to enhancing receptivity to instruction through facilitating positive 
academic, social, emotional, and physical development. It is clear that attending to such functioning is 
basic to preventing, treating, and remedying problems. Moreover, the goals and objectives provide 
direction for daily program planning and for evaluation. 

The assumption in pursuing goals and objectives is that optimal processes (comprehensive and integrated 
programs) will be used to create a match that enhances positive attitudes, growth, and learning. This 
applies to the full range of support available to students and families ~ including specialized programs at 
the site, home, and community. Until a comprehensive, integrated continuum of programs and services 
are in place, steps must be taken to address the less than optimal conditions. From this perspective, 
evaluation focuses on (a) individual student outcomes (related to the goals and objectives set forth in 
Exhibit A) and (b) outcomes for all children in the catchment area (e.g., community indicators of improved 
health, safety and survival, emotional health, and positive social connections). In addition, there can be 
a focus on outcomes reflecting significant changes in support systems (e.g., measures of enhanced home 
involvement in schooling; indicators of enhanced integration of center and community health, social, and 
mental health services -- including related data on financial savings). 

Furthermore, in pursuing goals and objectives related to instructional receptivity and social-emotional and 
physical development, it is essential to do so in ways that value and foster rather than devalue and inhibit 
appropriate diversity among students. This is especially important given the diversity students bring with 
regard to ethnic background, gender, interests, and capabilities. Thus, another focus for evaluation is on 
these concerns (especially in assessing for negative outcomes). In particular, efforts should be made to 
measure (a) movement toward inappropriate conformity in thinking and behaving in areas where diversity 
is desired and (b) trends toward increased levels of other-directedness and excessive dependency. 

(2) Outcomes related to preventing and correcting emotional, behavioral, learning, and health 
problems. In addition to the above goals and objectives, student goals and objectives are formulated in 
connection with specialized programs designed to prevent and correct emotional, behavioral, learning, and 
health problems. These objectives relate to the efforts of such programs to remove barriers and enable 
students to pursue the above goals. 



It is important to emphasize that problems become of concern because they are reflected in the student's 
functioning; however, the primary source of the problem often is environmental. Environmentally based 
problems are an especially important focus for prevention programs. Such programs are targeted to 
designated at-risk populations (e.g., students with older siblings in gangs, immigrant and highly mobile 
families who have major transition and school adjustment needs, students who experience a crisis event). 

In general, then, immediate objectives in working to address emotional and behavioral problems with a 
view to enabling student progress often include activity designed to reduce specified barriers to school 
attendance and functioning. Thus, attending to mental health concerns often requires addressing practical 
deterrents such as health problems, lack of adequate clothing, problems in the home, working with home 
to increase support for student improvement, dealing with student's physical or sexual abuse, dealing with 
student s substance abuse, dealing with gang involvement, provisions for pregnant minors and minor 
parents, dropout outreach and recovery, teaching student to use compensatory strategies for learning, and 
so forth. And, based on the discussion to this point, hopefully it is clear that the first indicators of progress 
may be fewer problems related to learning, behavior, and affect. See Exhibit A for examples of key 
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intervention goals and objectives and potential indicators of efficacy. The goals and objectives listed in 
Exhibit A represent individual student outcomes that can be measured as indicators of the impact of 
specialized programs. Positive "side effect" outcomes worth measuring are significant changes related to 
(a) all children in the catchment area (e.g., community indicators of improved health, safety and survival, 
emotional health, and positive social connections) and (b) support systems (e.g., enhanced home 
involvement in schooling; enhanced integration of a school-based health center and community health, 
social, and mental health services -- including related data on financial savings). Of course, additional 
student outcomes can be delineated and measured with respect to efforts to prevent specific types of 
problems. This is usually accomplished by fostering positive functioning through activities designed to 
enhance knowledge, skills, attitudes, and action related to healthy physical and mental development. Some 
of these efforts are carried out in special settings, such as school-based health centers and family resource 
centers. Whether or not there is a special setting, these efforts include specialized programs focused on 

• home involvement to enhance social-emotional development 

• peer-to-peer interventions designed to enhance social-emotional development 

• early education for prenatally drug-exposed children and their families 

• substance abuse prevention 

• suicide prevention 

• physical and sexual abuse prevention 

• violence prevention 

• dropout prevention and school re-entry 

• STD/AIDS prevention 

• pregnancy prevention 

• prenatal care of pregnant minors and minor parent education 

• crisis intervention and emergency responses to prevent long-term impact (e.g., PTSD) and to prevent 

subsequent emergencies K 

Intended Impact on Families and Community 



Aims related to families encompass promotion of positive family development and functioning and enhanced home 
involvement in schooling. Aims for the community encompass promotion of positive community development 
and functioning and related reform of community agencies (with particular emphasis on reducing problems related 

to health and safety). See Exhibit B for examples of key intervention goals and objectives and potential indicators 
of efficacy. 



Intended Impact on Programs and Systems 



Major aims with respect to the school-site are to promote and support (a) a major restructuring of school support 
services, (b) integration of school support services with other school-based/linked support programs, teams, and 
special projects (in both the regular and special education arenas), (c) outreach to enhance linkages and collaborations 
with community resources (e.g., health, social, recreational programs; involvement of volunteers and local 
businesses), and (d) integration of all activity designed to address barriers to learning with the instructional and 
school management components. See Exhibit C for examples of key goals and objectives and of potential indicators 



The Center is co-di reeled by I toward Adelman and Linda Taylor and operates under (he auspices of the Schoc 
Mental Health Project. Dept, of Psychology. UCLA. Los Angeles, CA 90095-1563 - Phone: (310) 825-3634. 
Support comes in part from the Department of Health and Human Services, PUS, URSA, 

Maternal and Child Health Bureau, Office of Adolescent Health. 
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Intervention Impact on Students 



<D 

cd V 

•c 5? 

<U G 
X O 

t-H J 
"O ' 

■i w 

S .2 
3 *3 
c/d 2 



C/} 

E 



Q m 
W h 

w S 55 

« S >. 

O CQ 
H W 
H 
W 

Q 



d 

d 

a* 



>» <2 

E S 

I -§ 



>> 

XI 



>> 

X 

co 

GO 

.s 

ca 

Oh 



O 

D- 

D 



D 

on 



co 

D 

a 

3 

c 

d 

a 

c 

ca 



d 

cu 



3 

c r 
d 

a> ^ 

.. -5*1 «* 

e M ” o 

° S J S 

co .G # > -o 

a « | I 

3 2 3 3 

t p • • 



f S 

*G co 
o u. 
cd Si 
um X 

QJ *-* 

> O 

^ o c 

>v GO ° 
X C <u 
3 e O 

x E c 

•a ** u 

CO O p ’O 

^ 2 1 s 

jj .2 B u 

2 T 3 TD •O 

• • • 



J2 



1 1 
cd cd 
X 
d 

CO Um 

£2 co .2 
00 JS > 
.E -q x 

3 2 a> 
« « X 
tS C co 
3 0-3 
a- a £ 



D 



cd 

1 1 ._ 

«A8 

a ■ ■ 



a 

,cd 



co 



CO 


■*—* 
4— * 

cd 


0 ) 

X 


E 


CD 


o 

cd 


o 


> 


a> 


Q. 

E 


cd 

GO 


«-*— T 


>> 


CD 


(D 


CO 


G 


CO 


• 


• 





D 

D 

o. 



o 

’cd* 

E 

cd O 

* 50 

D 73 
co C b 
i— i fd O 
3 0 

O co ^ 

<J O 

, 73 co 

bs cd D 
O u, «-* 

r. GO W 

C § 

a 

cd co oj 

S "t > 

53 co 5) 

S 3 *s 

X o x 

D O o 
w X cd 



d 



C« .S' 



d <u 
oo a 
73 c 

D cd 

> i 

O D 

5 2 

a> co 
co d 
cd 73 

a a 

<j • — 

£ 13 



X> 

'i . 

8. g> 

co X 

<U 73 
u C 

Ct— £D 

o 3 

iU ^ 
2 

§ •£ 
D.'g 

D X 
U <J 

8 .S 

• w 



«a 5 & 

co 55 C 
G =G GO 
O ^ D _ 
•— D *3 GO 

3 p .2 e 

.!= c ^ 

■o o C 5 



c 

o 

co • — 



. 2 “ cd 

.£ « «y ■§ •§ 

S> CJ c /5 C s 

5 <L> S X o S 

o <u T .2 x . i 

— 7 W-, rr 3 a— i 



c g 1 
•2 c 
3 .2 
3 3 
oo c 

a a 
a - 
.2 s 



OJ 

u 

c 

a 

<u u. 

X o 



o 

X 



X .a X 

n co 



—H 57- X u (d 3 

O CO CD O — < <u 

^ co co P co 



<j co 

<D >i 

b x 
73 ?■ 



> 
cd 
X 
0> 
X 

cd 

2 ^ 

X co 

• • 



0> 

CJ 

3 

■8 
Q S 





o 


X 


73 


GO 


a> 


C 


4— > 

cd 


‘E 


CD 


o 


Um 




CO 


a 

c 


E 


a 


a 




X 


73 


o 


C 


u. 


cd 


CL 


a> 


GO 


<j 


C 


c 

cd 

73 


CO 

CO 

D 


G 


Um 


(D 


73 


W 


73 


c3 


cd 



.I 1 

X ^ 

a js 

cd g - 
3 *3 

cr g 

CD 2 
73 O 

-1 

° £ 
rr co 



O 

O- 

Q- 

3 

co 

0> 

E 

o 

X 



o 
To 



u 

X * ’G i2 

• • • • 









O 












G 




>» 


w D 

C co 






E 




Um 

o 


D 3 
C X 

C rrt 

D 

11 


D 

CO 

3 


C 

D 

E 


GO 

c 

c 

D 




Cd rrj 

c .2 

D GO 
Dm D 


§• 2 


X 

cd 


D 

> 


Um 

cd 

o , 




E cd 
o b 


C co 


D 


O 






O CO 


b ^ 

C CJ 
fli • — 

■8 5L 


o 

C 

cd 

w 

co 


> 

c 

GO 


C 

cd 

G 

GO 


3 £ 00 

3 £ C 

o x 

Dm 73 F 


3 x 


X 

3 


G 

cd 


D 

Um 


O 

Um 


D ^ 

a H 


CO Om 


co 


GO 


Dm 73 


• 


• 


• 


• 


• 


• 



a* 

<u 

a 

aj 

Um 

a> 

u 

e 

cd 



E 

ii 

X 

O 

* Uh 

W Cm Om 
8 | 5 
g|« 

O X <D 
<j cd J3 



73 

e 



s i 



GO 

c 



•§ 2 
w ,S 



c o 

(D • — 

s | s 
£ 5 « 



o 

ERLC 



o 

•fH 



Intervention Impact on Families and Communities 

Aims Examples of Examples of Indicators Standards/Criteria 

Goals/Objectives of Efficacy Immediate -- Long-term 
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Intervention Impact on Families and Communities 

Aims Examples of Examples of Indicators Standards/Criteria 

Goals/Objectives of Efficacy Immediate -- Long-term 
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Intervention Impact on Programs and Systems 

Aims Examples of Examples of Indicators Standards/Criteria 

Goals/Objectives of Efficacy Immediate — Long-term 
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Center f A Brief Discussion of: 

Evaluation, Accountability, and Mental Health in Schools 

How effective is the intervention? 

Do you have data to support that approach? 

Where ’s your proof? 

The questions are so logical and simple to ask, and they can be so devastating in their imoact 
The problem is that such questions imply that relevant data are easy to gathef and so if data ' 
aren available the intervention must be ineffective or else those in charge ar’eTrresponsibTe. 

sua y ignored by the questioners are the many complexities associated with valid and ethical 
evaluation of major mental health and psychosocial problems. 

Every mental health practitioner is aware of the importance of having data on results All 
interveners want to be accountable for their actions and outcomes. But it is complicated. 

Fundamental dilemmas stem from the limited validity and focus of available measures and the 

aniH nCy f ° r th ° Se demand,n S accountability to have inappropriate expectations that there can be 
rapid improvement even though youngsters and their families are experiencing severe and 
pervasive problems. Most widely sanctioned evaluation instruments are quite fallible Moreover 
they are designed to measure results that require a lengthy course of intervention thereby giving 
short shrift to immediate benefits (benchmarks) that are essential precursors of longer-range § 
improvements. Ironically, demands for accountability tend not to fake responsibility for the 
negative consequences that formal assessment has on some clients. Accountability pressures 

diern^mfnv ^ly the gathering of a significant amount of data during the first session with a 
client, many practitioners note that this practice interferes with building positive relationshins 
and contributes to what is already too high an intervention dropout rate. 

What are practitioners and program leaders to do? 

Well, not surprisingly, they often look for assistance. The topics of evaluation, accountability 
and quality improvement are among the most frequent requests for technical assistance and 
continuing education. As a result, the number of publications and tecImSiScSurces 
n the area has increased at an exponential rate. And, there are endless lists 
t t have not been appropriately validated). Unfortunately, the volume of materials and other 
resources is not an indication that fundamental evaluation concerns have been effectively 
addressed. The complications remain unresolved, the status quo remains unsatisfactory and all 
that any of us can do at this point is to develop aids, guidelines, and staKC^kStoS 
strive for appropriate accountability while doing the least harm to youngsters Ld fheir families. 

“ a jJ L°ioMi S li" V ° IV K d W J th menta J health in schools ’ our ‘"tent here is to support evaluative 

ro“aih e eisrsr i,y ,o ,he society ’ ,o L ins,i,u,io " such 2 ^“sr s for 



o 

ERIC 



is “-directed by Howard Adelman and Linda Taylor and operates 
under the auspices of the School Mental Health Project, Dept, of Psychology, UCLA 

Website: http://www.lifesci.ucla.edu/psych/mh/ 

c*" f rom the US. Department of Health and Human Services, 
Public Health Service, Health Resources and Services Administration, 
Maternal and Child Health Bureau, Office of Adolescent Health. 
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Accountability to Who? 



In a seminal article on the evaluation of therapeutic outcomes, Strupp and Hadley (1977) stress 
how different the expectations of society and its institutions often are from those of individual 
clients. 1 Thus, it is imperative to understand accountability from the perspective of the various 
parties who have special interest in the results of interventions that are meant to address mental 
health and psychosocial concerns. For our purposes here, the focus is on (a) the society in general 
and the institution of schooling in particular and (b) those specific youngsters and their families 
who are the direct focus of intervention efforts. 



Accountability to Society and to the Institution of Schooling 

Society looks at the following general indicators to evaluate whether efforts related to psychosocial 
and mental health concerns are paying appropriate dividends: 



Increases in youth employment (ages 16-19) 
Reductions in 

• student mobility 

• youth pregnancy 

• sexually transmitted diseases 

• child abuse/neglect 

• youth arrest/citation 

• youth probation violations 



Reductions in 

• youth emergency room use for mental 
health and psychosocial related events 

• foster care placements 

• homeless youth 

• youth suicide rates 

• youth death rates 



In addition, those responsible for schools are required to demonstrate effective fulfillment of their 
specific mission — which is to educate the young in ways that meet society’s needs. The primary 
indicators currently demanded by social policy are those that reflect academic achievement at a 
standard that is competitive with other major countries. Thus, the emphasis on increases 



at all grades with respect to 

• achievement test scores 

• grades 

• other indicators of progress 

in academics (analyses of work) 



at high school level with respect to 

• number graduating (with a related 
reduction in the number dropping out) 

• number taking SATs 

• number continuing with post-secondary educ 



Because many youngsters are experiencing barriers to learning and performing at school, programs 
and services to address such barriers are increasingly essential to the ability of schools to 
accomplish their mission. Some major indicators for accountability related to these enabling or 
learning support programs are 



Reductions in 

• unexcused absences 

• tardies 

• suspensions/expulsions 

• referrals for misbehavior 

• referrals for learning problems 



Increases in 

• attendance 

• cooperation & work habits 

• fluency in English as Second Language 
Reduction in numbers designated as Learning 

Disabled or Emotionally Disturbed 



As these writers state: “Society is primarily concerned with the maintenance of social relations, institutions, 
and prevailing standards of sanctioned conduct. Society and its agents thus tend to define mental health in terms of 
behavioral stability, predictability, and conformity to the social code. ... The individual client ... wishes first and 
foremost to be happy, to feel content [and) thus defines mental health in terms of highly subjective feelings of well- 
being ... (p. 188). Strupp, H.H. & Hadley, S.M. (1977). A tripartite model for mental health and therapeutic 
outcomes with special reference to negative effects in psychotherapy. American Psychologist, 32 % 187-196. 
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Data for Accountability to Society and the Institution of Schooling 

Data related to most of the above indicators are available from the records at school sites 
school districts, and city/county agencies. Some schools also are involved in administering 
the Youth Risk Behavior Surveillance System (sponsored by the Centers for Disease Control 
and Prevention) which contains relevant indicators for use in monitoring changes over time 
(Many communities and child advocacy groups are gathering local and statewide data on 
child well-being and publishing it as "Report Cards.") If data are not available, then efforts 
are needed to ensure relevant indicators are gathered and made accessible. And, appropriate 

subgroups be taken t0 6nSUre data Can be disa 88 re 8 ated with res P e ct to specific 



Accountability to Specific Youngsters and Families 

Those who work in school districts to provide programs and services related to psychosocial and 
mental health concerns also are accountable to the specific individuals they help. Such 
accountability certainly can be seen as encompassing the indicators listed above. However for 
individuals who must deal with major barriers, many of the above realistically are only good 
indicators of progress after a lengthy period of multifaceted, comprehensive integrated 
intervention. More immediate accountability indicators are needed to demonstrate progress related 
to objectives that are the current and direct focus of psychosocial and mental health interventions 
(e.g., reductions in symptoms; enhanced motivation and psychological and physical well-being). 
Because data on such specific objectives are not readily available, the problem of generating 
relevant data arises -- as do some serious dilemmas. Efforts to answer the following questions lead 
to an appreciation of the many problems and issues. 



What are the right indicators? 

Endless arguments arise over indicators when they are discussed in highly specific 
and concrete terms. At a more abstract level, there is considerable agreement ‘ 
around three general categories: (1) client satisfaction (the youngster; the family) 
(2) reduction in the youngster's symptoms/problem behaviors, and (3) increases in 
positive functioning (the youngster; the family). 



How can appropriate specific and concrete indicators he identified for particular clients? 

The dilemmas that arise here reflect the problem of "Who is the client?" - the 
youngster? the family? a teacher who made the referral? Additional dilemmas 
arise because the various involved parties often have different perspectives 
regarding what problems should be addressed. (And , of course, the intervener 
may have even another perspective.) A reasonable compromise is to gather 
evaluative data related to (1 ) the specific symptoms and behavior problems that 
led to the referral, (2) any objectives that the client wants help in achieving, and 
(3) specific objectives that the intervener believes are warranted and that the 
client consents to add. 
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How should the deficiencies associated with existing measures be accounted for? 

Although some measures are better than others and some are designated the best that 
exist, best should not be equated with good or good enough. All instruments we rely 
on currently have limited reliability and validity; also quite limited are the normative 
data for various subgroups. These limitations (1) call for using formal instruments 
only when they are necessary, (2) require full disclosure of limitations in reporting 
findings, and (3) warrant making extreme efforts to look for disconfirming evidence 
whenever findings suggest significant pathology. 



How can the negative impact of gathering the data be minimized to an appropriate degree? 

All evaluation has the potential to produce major negative consequences. The ethical 
obligation is to maximize benefits and minimize costs to clients. Putting aside the 
financial costs, it is clear that use of any formal measure can increase a client's 
distress and produce psychological reactance. It is likely that the high dropout rate 
among clients, in part, is a reaction to too much formal assessment during the first 
encounters with an intervener. Accountability requirements that mandate 
administration of formal measures before counseling is initiated may well be 
contributing to the low rate of youngsters who stay in counseling long enough to 
reap significant benefits. From the perspective of sound standards for practice, (1) no 
formal measures should be administered until the intervener judges that the 
relationship with the client is strong enough to mediate any distress and (2) measures 
should be personalized to assess only the specific and concrete indicators that are 
relevant to a particular client. 



Accountability: Is it Becoming a Mantra? 

Accountability should not simply be a 
mantra. It is an invaluable facet of 
effective practice; but it is just one facet 
and only makes sense when the other 
facets are properly planned and 
implemented. 
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Measures Relevant for Accountability to Specific Youngsters and Families 

Below are listed a sample of promising instruments. Unless otherwise noted, the measure cited is reviewed 
m Evaluating the Outcome of Children 's Mental Health Services: A Guide for the ^SJS£kcSSl 

A t5? mily r Ut ?° m c e M f a t ure l s (1.995) - prepared by T.P. Cross & E. McDonald for the Technical 
Assistance Center for the Evaluation of Children’s Mental Health Services. 2 

It is essential that interveners review and choose measures that minimize negative impact on clients. 
Proper personalization of assessment in the best interests of the client may even call for not using a 

— m f . ltS ei J. tir f y . or In the way the developer prescribes. We recognize that this violates 
standardization of administration and makes interpretation more difficult, but just as empirically 

nrarti^ he £ a P^ utlc strategies must be adapted to ensure a good fit with a client, so must assessment 
practices. In both instances, empirical support for prevailing practices is not so strong as to warrant 
rigid implementation. Also of value are data from functional assessments (increasingly being done 
w en smdents are referred for behavior problems). Finally, some interveners use projective procedures 

PhilHrpnfr? ' temS fr< ? m other , measures ( e -g-> sentence completion, drawings and related stories, 
Childrens Depression Inventory) as a stimulus for discussion with clients. Client responses early and 
near the end of the period of intervention may be useful as supplementary evaluation data. 

(1) Client Satisfaction (youngster; family) 

Client Satisfaction Questionnaire (CSQ — Larsen, et al. - Portland State U. Version) 

Youth Satisfaction Questionnaire (YSQ - Portland State U.) 

Vanderbilt Satisfaction Scales - parents/caregivers and/or adolescent self-report 

(2) Reduction in Youngster's Symptoms/Problem Behaviors 

Child Behavior Checklist (CBCL - Achenbach & Colleagues) 

There are versions to be filled out by parents-caregivers, teachers, and youth self- 
report, as well as a direct observation form. 

Child Assessment Schedule (CAS - Hodges) - self-reports from child and/or parents-caregivers 

(3) Increases in Positive Functioning (youngster; family). 

Child and Adolescent Functional Assessment Scale (CAFAS - Hodges) - intervener rating 
Preschool and Early Childhood Functional Assessment (Hodges) - intervener rating 
Quality of Well-Being Scale 1 (QWB) - client self-report 
Family Environment Scale (Moos) - family self-report 

Family Empowerment Scale (Portland State U.) - family self-report 



-Other instruments are reviewed in the guidebook; those included here seem most useful for practitioners 
concerned with mental health in schools. The guidebook is available by contacting the TA Center for the 

eSr MA Of 02 f W2%T h SerViCe$ at JUdge Baker Children ' S Center ' 295 Lon ^ ood Ave, 

T Hu - & B c “ na 
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Sampling of Indicators with Respect to Different Accountability Demands 

As should be evident from the preceding discussion, it can be extremely costly and time 
consuming to be accountable to all parties (see also Figure 1) with interests in the productivity of 
an intervention. In most situations, the reality is that only a sample of data can be gathered (see 
Figure 2). 

With respect to individual clients, the data sample should begin with assessment that has direct 
and immediate relevance to the specific objectives an intervener and client have agreed to pursue. 
Then, in response to accountability demands and in keeping with ethical and feasible practice, a 
subset of standardized items can be administered to stratified samples of clients. The particular 
subsets of items chosen should reflect matters of greatest concern to those demanding 
accountability. If the pool of items is large, then different subsets of items can be administered 
over time and later combined to provide a full picture of outcomes. 

With respect to societal and institutional accountability, the data sample initially consists of that 
which can be readily gathered on a regular basis. Subsequently, again reflecting matters of greatest 
concern to those demanding accountability, step by step strategies can be developed to establish 
systems for amassing regular findings related to key variables and specific population subgroups. 

Clearly, sampling requires considerable planning and careful implementation. A systematic 
evaluation plan must be developed, and there must be appropriate budgeting for its 
implementation. Many programs will require specific consultation in developing an appropriate 
sampling strategies. 



Standards for Comparison 

Whatever data are collected will be imperfect and only rarely will be easily interpreted. For 
accountability to be rationale, there must be a reasonable set of standards for comparison. In 
asking how good an intervention is, the question must be answered in terms of Compared to what? 



When it comes to mental health in the schools, the best comparisons are (a) data on the previous 
results of intervention efforts with comparable students and their families, (b) data on similar 
students/families at a school who have not yet been served (e.g., appropriate waiting list samples), 
or (c) data from a very similar school that does not have the programs being evaluated. The first 
approach calls for gathering a “baseline” of data before or in the early stages when an intervention 
is being developed. The latter approaches call for being able to gather the same data with 
nonserved groups. Again, the matters of systematic planning and appropriate budgeting are central 
considerations. 



Finding out if interventions are any good is a necessity. But in doing so, it is wise to 
recognize that evaluation is not simply a technical process. Evaluation involves decisions 
about what, how, and when to measure, and these decisions are based in great part on values 
and beliefs. As a result, limited knowledge, bias, vested interests, and ethical issues are 
constantly influencing evaluation processes and the decisions made with respect to 
accountability. 



Figure 1. Accountability as related to differing intervention goals. 



Purpose of Intervention 


Accountability to Who? 


Sampling of 
Objectives/Goals 


To meet society ’s goals 


Accountable to society 


Data are gathered on 
indicators that reflect 
society’s purposes in 
financing the institution 


To meet an institution ’s 
goals 


Accountable to a 
specific institution 


Data are gathered on 
indicators that reflect the 
institution’s purposes 


To meet the personal goals 
of specific clients 


Accountable to 
specific clients 


Data are gathered on 
indicators that reflect 
individual client’s purposes 
in participating in an 
intervention 


To meet some combination 
of society, institution, and 
individual goals 


Combination of the above 


Combination of the above 
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Figure 



Note: 
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2. Evaluation/accountability/results: Sampling indicators with respect to 
different accountability demands. 



Accountabili ty to Society 

: - (to meet society’s goals|||J| 



M 

: : :w: ; 



! MU 



Increases in 

>youth employment (ages 16-19) 

>readiness for adult employment 
Reductions in 

>youth pregnancy 

>arrests/citations/probation violations 
>Sexually transmitted diseases 
>child abuse/neglect 

>emergency room use for mental health/psychosocial problems 
>youth foster care placements/homelessness 
>youth deaths 



m n 
H u 



I a: 



A ecoantabilit| to a Specific ^ 

(«> meet the institution’s goals) 
e.g., schools 
Increases in 

>academic achievement and grades 
>graduation rates 

>numbers taking college board exams 
>numbers continuing on with post-secondary education 
>attendance/decreased tardies 
Reductions in 

>referrals for misbehavior/learning problems 

>numbers designated Learning Disabled/Emotionally Disturbed 

>numbers of dropouts 



Accountability tolSpecific O icn t(s) 

(to meet the personal goals of clients) 

>satisfaction with intervention 

>progress in addressing problems for which intervention 
was implemented (e.g., symptom reduction, increase in 
positive functioning) 



Not included here are indicators of negative effects that may accrue for individuals, interveners, 
institutions, and the society (e.g., the many psychological, social, and economic costs). Clearly,' 
data on these matters is essential - although they tend to be ignored in many so-called re.su/ts- 
oriented demands for accountability. 
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Evaluation and Accountability 



II. A Quick Overview of Some Basic Resources 



A. Selected References 

1. Developing Vour Rationale 

Evaluator’s Handbook: Vol.l 9160 pgs.), In Program Evaluation Kit. 

J.L. Herman, L.L. Morries, & C.T.Fitz-Gibbon (1987). Thousand Oaks, CA- Sage 
(Phone: 805/499-9774). 

New Approaches to Evaluating Community Initiatives - Concepts. Methods and Contexts. 

J. P. Connell, A.C. Kubisch, L.B. Schorr, & C.H. Weiss (Eds.). Washington, DC: The 
Aspen Institute, 1995. 

On Understanding Intervention in Psychology and Education. H.S. Adelman & L. Taylor. 
Westport, CT: Praeger, 1994. See section on “The Evaluation Problem”. 

Multi-Stage Evaluation for a Community Mental Health System for Change. 

K. Kutash, A. Duchnowski, M. Johnson, & D. Rugs (1993). Administration and Policy 
in Mental Health , 20, 31 1-322. 

An Interorganizational Network Approach to Evaluating Children’s Mental Health Service 
Systems. 

J.P. Morrissey (1992). New Directions for Program Evaluation, 54. 85-99. 

The Process Study Components of Mental Health Evaluation. 

R.D. Perry, B.H. Hoff, & D.S. Gaither (1994). Evaluation and Program Planning. 17, 



What Works for Whom: The Design and Evaluation of Children’s Mental Health Services. 
M. Armstrong, S. Huz, & M. Evans (1992). Social Work Research & Abstract, 28, 35- 

Child and Adolescent Mental Health Services: Evaluation Challenges. 

L. Bickman & D.J. Rog (1992). Evaluating Mental Health Services for Children . 54, 5- 
16. 

Quality. Quality Assessment, and Quality Assurance: Considerations for Maternal and 
Child Health Populations and Practitioners. 

H. Grason& B. Guyer(1995). Baltimore: The Child and Adolescent Health Policy 
Center, The Johns Hopkins University. 
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2. Specific Topics 



Evaluating Children’s Mental Health Systems: An analysis of Critical Behaviors and Events. 

D. Rugs & K. Kutash (1994). Journal of Child and Family Studies, 3, 249-262. 

Scales to Assess Child and Adolescent Depression: Checklists, Screens and Nets. 

E. J. Costello & A. Angold (1988). Journal of American Academic Child and 
Adolescent Psychiatry, 27, 726-737 . 

Structured Interview for Assessing Children. 

K. Hodges (1993). Journal of Child Psychology and Psychiatry, 34, 49-68. 

Improving Accountability in a Service Delivery System in Children’s Mental Health. 

J.D. Burchard (1992). Clinical Psychology Review, 12, 867-882. 

Assessing Mental Health Outcome Measurement Techniques. 

J. A. Ciarlo, T.R. Brown, D.W. Edwards, T.J. Kiresuk, & F.L. Newman (1986). 

National Institute of Mental Health. Series FN No. DHHS Pub. No. (ADM) 86-1301. 
Washington, DC: Supt. of Docs., U.S. Govt. Printing Office. 

Assessing Outcomes for Sufferers of Severe Mental Disorder: A Conceptual Framework and 
Review. A. Rosenblatt & C.C. Attkinsson (1993). Evaluation and Program Planning 
16, 3347-3363. 

Measuring Treatment Outcome and Client Satisfaction Among Children and Families. 

Special Section: Outcomes research. 

T.G. Plante, C.E. Couchman, & A.R. Diaz (1995). Journal of Mental Health 
Administration, 22, 261-269. 

Cost-Outcome Methods for Mental Health. (242 pp.) 

W. Hargreaves, M. Shumway, T. Hu, & B. Cuffel (1998). San Diego, CA: Academic 
Press. 

Integrating Systems of Care in California for Youth with Severe Emotional Disturbance 
(Two reports from the California AB377 Evaluation Project). A. Rosenblatt & C.C. 
Attkinsson (1992). Journal of Child and Family Studies, l, 93-113; 263-286. 

Use of Structured Assessment Tools in Clinical Practice. M.D. Weist, & M. Baker (1995). 
Paper presented at the 1995 annual meeting of the American Society for Adolescent 
Psychiatry. 

Analyzing Costs, Procedures, Processes, and Outcomes in Human Services. 

B.T. Yates (1996). Thousand Oaks, CA: Sage. (161 pp.) 

Outcomes and Evaluation: System, Program and Clinician Level Measures. 

S. Essock & H. Goldman (1997). In K. Minkoff & D. Pollack (Eds.), Managed Mental 
Health Care in the Public Sector: A Survival Manual. Singapore: Harwood Academic 
Publishers, (pp. 295-307). 



Evaluating Family Programs. (556 pp.) 

H.B. Weiss, & F. H. Jacobs (1988). Hawthorne, NY: Aldine de Gruyter. 



B. Guidebooks and Models 

Evaluating the Outcome of Children’s Mental Health Services: A Guide for the Use of Available 
Child and Family Outcome Measures (1995) - by T.P. Cross & E. McDonald 

Discusses ways to use available standardized child and family outcome measures in the 
development of an outcome measurement plan. Describes the process used to select a standard 
instrument, discusses criteria used as the basis for instrument selection; presents basic information 
on a selected set of instruments suitable for use in measuring child and family outcomes. 

To order a copy of this guide, contact: 

The Technical Assistance Center for the Evaluation of Children’s Mental Health Systems/ 
Judge Baker Children’s Center; 295 Longwood Ave., Boston, MA 02115 
Phone: (617) 232-8390/ Fax: (617) 232-4125. 



Center for the Study of Evaluation: Program Evaluation Kit (1987) 

This nine volume kit offers a step-by-step guide to planning and conducting program evaluations. 
Titles include. Evaluator s handbook , How to focus on evaluation , How to design a program 
evaluation, How to use qualitative methods in evaluation, How to assess program implementation, 
How to measure attitudes, How to measure performance and use tests, How to analyze data, and 
How to communicate evaluation findings. 

Contact: Sage Publications, Inc., P.O. Box 5084, Thousand Oaks CA 91359-9924 
Telephone: (805) 499-9774; Fax: (805) 499-0871 ; Internet: order@sagepub.com 



National Institute on Drug Abuse 

How Good is Your Drug Abuse Treatment Program? A Guide to Evaluation (1993) - by the National 
Institute on Drug Abuse (NCADI #BKD104) 

This guide is highlighted in Section IV of this Introductory Packet. The guide outlines a 52 week 
evaluation plan, and the steps necessary to meet each week’s goals. This model encompasses 
developing a program plan, with concrete objectives and goals; organizing resources; 
operationalizing measures; developing a research design; collecting and analyzing data; and finally 
reporting and using the findings. Also outlined is how to use the developments from earlier steps 
as stepping stones to later ones. Although this model pertains to drug treatment programs, the 
basic strategies can be applied to various intervention programs. 

Contact: The National Technical Information Service order desk, 5285 Port Royal Road 

Springfield, VA 22161. Phone: 703-487-4650; FAX: 703-321-8547 (To verify receipt of 
your fax, call 703-487-4679). For RUSH service: 1-800-553-NTIS 
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A. Rosenblatt & C. C. Attkinsson (1993). Assessing Outcomes for Sufferers of Severe Mental Disorder: 
A conceptual Framework and Review. Evaluation and Program Planning, Vol. 16, pp. 347-363. 

A brief summary of this framework is presented in a box on the next page. This article presents 
a conceptual framework to classify the outcomes of services (and thus outcome measures). The 
classification framework integrates three dimensions: 1) respondent type, which reflects a range 
of social perspectives: client, family, social, clinician, and scientist; 2) social context of 
measurement, which states that measures must be taken in the context of all areas of functioning: 
individual/self, family, work/school, community; 3) treatment outcomes, is based on the need for 
multiple measures and approaches to measuring outcomes for persons suffering from severe mental 
disorders. 

K. Hoagwood, P.S. Jensen, T. Petti, & B.J. Bums (1996). Outcomes of Mental Health Care for 
Children and Adolescents: I. A Comprehensive Conceptual Model. Journal of the American 
Academy of Child and Adolescent Psychiatry, 35. 

Outlines a dynamic and interactional model of outcomes that braodens the range of intended 
consequences of care. It comprises five domains: Symptoms, functioning, consumer perspectives, 
environmental contexts and systems. The model reflects the changeable interaction between 
children’s evolving capacities and their primary environments (home, school, and community). 



C. Agencies and Website for Technical Assistance 

In addition to our Center and the Center for School Mental Health Assistance (University of Maryland at 
Baltimore) - which provide technical assistance support and put out a variety of publications - the 
following agencies can also be of assistance. 

ERIC -- Clearinghouse on Assessment and Evaluation 

The Educational Resources Information Center (ERIC) is a national information system. One of 
its divisions focuses on assessment and evaluation, providing access to technical assistance services 
as well as documents and reports, test banks, and more. Their web site provides links to relevant 
sites, and information on many other ERIC programs and services. 

Contact. O Boyle Hall, Department of Education, The Catholic University of America 

Washington, DC 20064; Website: http://www.cua.edu/www/eric_ae/MAIN.HTM 

Harvard Family Research Project 

Focuses on family support programs and policies; provides technical assistance to a nationwide 
network of practitioners, policy makers, and educators. Publishes "The Evaluation Exchange," a 
quarterly newsletter; their website links to agencies, foundations and think tanks involved in child 
and family issues and research. 

Contact: Harvard Family Research Project, 38 Concord Avenue, Cambridge, MA 02138 
Phone: (6 1 7) 495-9 1 08 Email: hfrp@hugse 1 .harvard.edu 

Website: http://hugse 1 .harvard.edu/~hfrp/ 
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National Center for Educational Outcomes (NCEO) 

Specializes in the identification of outcomes, indicators, and assessments to monitor educational results 
or all students including students with disabilities. Has an extensive publication list, a directory of 
assessment projects, a national network of technical assistance providers. 

Contact: University of Minnesota, 350 Elliott Hall, 75 East River Road 

Minneapolis, MN 55455 Phone: (612) 626-1530 Fax: (612) 624-0879 
Website: http://www.coled.umn.edu/nceo/ 

Technical Assistance Center for the Evaluation of Children’s Mental Health Systems 

This center is highlighted on the next page. Located at Judge Baker Children’s Center, this agency 
provides consultation and has a library of measures, manuals, and articles, including Evaluating the 

Outcome of Children’s Mental Health Services: A Guide for the Use of Available Child and Family 
Outcome Measures. 

Contact: Christina Crowe, Director, 295 Longwood Ave., Boston, MA 02115 
Phone: 617/232-4125 or (800) 779-8390 Fax: 617/232-4125 
Website: http://tac.pie.org/T3632 



Internet websites can be goldmines of information. They have reports, publications, online resources (e.g. 
catalogs, technical assistance), model programs, and links to other resources. 

Assessment and Evaluation on the Internet 
Website: http://ericae2.educ.cua.edU/intbod.stm#AA 

Developed by The Educational Resources Information Center (ERIC) to provide online services 
and documents pertaining to assessment and evaluation; contains information on special issues in 
evaluation, test descriptions, lists of online test publishers, and much more. 

The Evaluation Clearinghouse 

Website: http://www3.sympatico.ca/gpic/evaIweb.htm 

Specializes in linking to useful information and organizations related to evaluation and assessment 

has online documents, information discussion groups, links to relevant evaluation organizations 
and think tanks, and more. 




25 



30 



Measuring Mental Health Outcomes 



A Brief Highlight: from Assessing Outcomes for Sufferers of Severe Mental Disorder: A Conceptual 
Framework and Review, by A. Rosenblatt and C.C. Attkisson, Evaluation and Program Planning, Vol 
16, pp. 347-363, 1993. 



In light of the challenges facing the field of outcome research related mental health services, a conceptual 
framework is presented to classify the outcomes of services for sufferers of severe mental disorders. 
This classification framework integrates three dimensions: (a) the respondent type, (b) the social 
context, and (c) the domain of treatment outcomes based on the need for multiple measures and 
approaches to measuring outcomes for persons suffering from severe mental disorders. 



The conceptual framework consists of five respondent types (who), four behavioral/social contexts of 
measurement (where), and four domains of treatment outcomes (what) which are graphically represented 
in Figure 1 : 



Respondent types - 

measures of outcomes must reflect a range of 
social perspectives:client, family, social, 
clinician, and scientist 



Behavioral/social contexts of measurement - 
measures must be taken in the context of all 
areas of functioning: individual/self, family, 
work/school, community 



Domains of treatment outcomes - 
measures should cover all domains: clinical 
status, functional status, life satisfaction & 
fulfillment, safety & welfare 





DOMAIN OF TREATMENT OUTCOME 



Figure I, A model of (he dimension! of outcome measurement for 
mental health services research. 



This conceptual framework is useful in classifying and evaluating the usefulness of outcome measures, 
for example, who provides the data for the measure, what is the relevant social context, and what is the 
domain of treatment outcome? 



O 

ERIC 
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A Highlighted Resource 



Technical Assistance Center for the Evaluation of Children’s Mental Health Systems (located at the Judge 
Baker Children’s Center in Boston, 295 Longwood Ave., Boston, MA 021 15). This is a premier resource for 
information on specific measures; the Center provides consultation and has a library of measures manuals 
and related articles: Phone (800) 779-8390, ext. 2139. Website - http://tac.pie.org/T3632. ’ 



In reviewing measures, they provide detailed information on various instruments. Examples include: 

1) Hodges' Child and Adolescent Functional Assessment Scale (CAFAS) which they indicate measures 
■ client functioning in general and in each of six psychosocial areas: "Role Performance (includes legal 

problems), Thinking, Behavior Toward Others/Self, Moods/Emotions, and Substance Abuse. For each 
subscale, service providers are asked to rate the client at the most severe level of dysfunction that has occurred 
at anytime in the last month. In addition, two scales have been added to assess resources of caregivers' capacity 
to provide for a) basic needs and b) family/social support." 

2) Achenbach's Child Behavior Checklist which measures Externalizing behavior (directed outward - poor 
behavior control, etc.) and Internalizing behavior (directed inward - anxiety, depression, etc.). Scores for 
problem areas distinguish a) withdrawn behavior, b) somatic complaints, c) anxious/depressed, d) social 
problems, e) thought problems, f) attention problems, g) delinquent behavior, h) aggressive behavior, and i) 

sex problems. Scores for Social Competence distinguish school, social, and activities. 

DeChillo's Client Satisfaction Questionnaire (CSQ) which is an 8-item scale that yields an overall 
satisfaction score and specific items that might help with quality improvement of services. 

Each review also provides technical information on the psychometric properties of the instrument and discusses 
other relevant matters. 



Among the measures they cite are: 



Parenting Stress Index 
Child Behavior Checklist 
Early Childhood Assessment Tool 
Family Evaluation Scale 
Self-esteem measures 



Cultural Competence Instrument 
Diagnostic Interview Schedule for Children 
Adaptive Social Behavior Inventory 
Adaptive Functioning Measure 
Youth Satisfaction Questionnaire 



O 
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D. CONSULTATION CADRE 



Professionals across the country volunteer to network with others to share what they know. Some cadre 
members run programs; many work directly with youngsters in a variety of settings and focus on a wide range 
of psychosocial problems. Others are ready to share their expertise on policy, funding, and major system 
concerns. The group encompasses professionals working in schools, agencies, community organizations, 
family resource centers, clinics and health centers, teaching hospitals, universities, and so forth. 

People ask how we screen cadre members. We don’t! It’s not our role to endorse anyone. We think it’s 
wonderful that so many professionals want to help their colleagues, and our role is to facilitate the networking. 
If you are willing to offer informal consultation at no charge to colleagues trying to improve sytems, programs, 
and services for addressing barriers to learning, let us know. Our list is growing each day; the following are 
those currently on file related to this topic. Note: the list is alphabetized by Region and State as an aid in 
finding a nearby resource. 

Central States 

Minnesota Missouri 

Gordon Wrobel Sandra Nichols 

Mental Health Consultant Missouri Dept, of Elementary 

Minnesota Dept of Children, Families & Learning Secondary Education 

830 Capitol Square Building P.O. Box 480 

St. Paul, MN 55101 Jefferson City, MO 65202 

Phone: 612/297-1641 

Fax: 612/297-7368 

Email: gordon.wrobel@state.mn.us 



East 



District of Columbia 
Ronda Talley 

425 Eighth Street, NW, #645 
Washington, DC 20004 
Phone: 202/393-0658 
Fax: 202/393-5864 
Email: rct.apa@email.apa.org 

Maryland 

Kristin Langlykke 

Maternal and Child Health Bureau 

Office of Adol. Health 

5600 Fisher Lane 

Rockville, MD 20857 

Phone: 301/443-4026 

Email: ncemch01@gumedlib.dml.georgetown.edu 

Maine 

Ellen Bowman 

LCPC-Clinical Counselor 

Maranacook Community School-Student 

P.O. Box 177 

Readfield, ME 04355 

Phone: 207/685-3041 



New York 

Christopher Cinton 
Project Director 

Bronx-Lebanon Hospital Center 
1650 Grand Concourse 
Bronx, NY 10457 
Phone: 718/960-1328 
Fax: 718/583-0460 

Pennsylvania 

Patricia Welle 

Student Services Coordinator 

School District of the City of Allentown 

3 1 South Penn Street 

P.O. Box 328 

Alllentown, PA 18105 

Phone: 610/821-2619 

Fax: 610/821-2618 

Rhode Island 

Robert F. Wooler 

Executive Director 

RI Youth Guidance Center, Inc. 

82 Pond Street 
Pawtucket, RI 02860 
Phone: 401/725-0450 
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Southeast 



Alabama 

Deborah Cleckley 

Director, Quality Assurance/Education 
Jefferson County Department of Health 
1400 6th Avenue, South 
Birmingham, AL 35233-2468 
Phone: 205/930-1401 
Fax: 205/930-1979 

Florida 

Howard Knoff 

Professor and Director 

School Psychology Program 

University of South Florida 

4202 East Fowler Avenue, FAO 100U 

Tampa, FL 33620-7750 

Phone: 813/974-9498 

Fax: 813/974-5814 

Email: knoff@tempest.coedu.usf.edu 

Georgia 

Peter A. Cortese 

Chief Program Development and Services Branch 
Center for Disease Control and Prevention 
Division of Adolescent and School Health 
4770 Buford Highway, N.E., MS-K3I 
Atlanta, GA 30341-3724 
Phone: 404/488-5365 
Fax: 404/488-5972 
Email: pac2@ccdashl.em.cdc.gov 

Louisiana 

Theresa Nash 

Administrative Supervisor of School Nurses 
New Orleans Public Schools 
Medical and Health Services Department 
820 Girod St. 

New Orleans, LA 70113 
Phone: 504/592-8377 
Fax: 504/592-8378 



North Carolina 
Bill Hussey 
Section Chief 
Dept, of Public Instruction 
301 N. Wilmington St. 

Raleigh, NC 27601-2825 
Phone: 919/715-1576 
Fax: 919/715-1569 
Email: bhussy@dpi.state.nc.us 

William Trant 

Director Exceptional Programs 
New Hanover County Schools 
1802 South 15th Street 
Wilmington, NC 28401 
Phone: 910/815-6935 
Fax: 910/815-6929 
Email: nhcsswrt@uncwil.edu 

Virginia 

Sally McConnell 

Director of Government Relations 

National Association of Elementary School Principals 

1615 Duke Street 

Alexandria, VA 22314 

Phone: 703/518-6263 

Fax: 703/548-602 1 

Email: sallymac@ix.netcom.com 



Southwest 



California 

Frank Binch 

Department of Health Services County of Los Angeles 
1200 N. State St. 

Los Angeles, CA 90033 
Phone: 213/226-8326 
Fax: 213/226-8320 

Email: 73267.2635@compuserve.com 



Howard Blonsky 

Coordinator, Beacon Schools 

San Francisco Unified School District 

1512 Golden Gate Avenue 

San Francisco, CA 94115 

Phone: 415/749-3400 

Fax: 415/749-3420 



BEST COPY AVAILABLE 
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Southwest (cont.) 



California (cont.) 

Jim Bouquin 
Executive Director 
New Connections 
1760 Clayton Rd. 

Concord, CA 94520 
Phone: 510/676-1601 

June Cichowicz 

Community Relations Director 

Crisis and Suicide Intervention of Contra Costa 

P.O. Box 4852 

Walnut Creek, CA 94596 

Phone: 510/939-1916 

Fax: 510/939-1933 

Lois Coleman-Lang 

Coordinator of Health Services / Tracy Public Schools 

3 1 5 East 1 1th Street 

Tracy, CA 95376 

Phone: 209/831-5036 

Fax: 209/836-3689 

Georgiana Coray 

CA Assoc, for School-Based & 

School Linked-Health Centers 
7956 Grape St. 

La Mesa, CA 91941 
Phone: 619/464-3988 

Alfredo Crespo 
Psychologist 

San Fernando Valley Child Guidance Clinic 
9650 Zelzah Ave. 

Northridge, CA 91325 
Phone: 818/506-1348 
Fax: 818/998-2726 

Kimberly Dark 

Community Program Coordinator 

Reachout to Families 

South Bay Union School District 

610 Elm Avenue 

Imperial Beach, CA 91932 

Phone: 619/575-5917 

Fax: 619/424-9607 

Sylvia Dean 

Coordinator of Psychological Service 
Los Angeles School District 
1 1380 W. Graham Place - Bldg. Y 
Los Angeles, CA 90064 
Phone: 310/444-9913 
Fax: 310/914-9760 



California (cont.) 

John Hatakeyama 
Deputy Director 

Children and Youth Services Bureau 

L.A. County Dept, of Mental Health, C&FSB 

505 S. Virgil Avenue 

Los Angeles, CA 90020 

Phone: 213/738-2147 

Fax: 213/386-5282 

Susan Lordi 

Administrative Project Director 

Los Angeles County Office of Education 

9300 Imperial Highway, Room 210 

Downey, CA 90242-2890 

Phone: 310/922-6377 

Fax: 310/922-6299 

Email: lordi_susan@mssmtp.lacoe.edu 

Michael Pines 
Consultant 

Div. of Career & Family Servs. L A Cnty. Office of Ed. 

9300 Imperial Highway 

Downey, CA 90242-2890 

Phone: 310/940-1683 

Fax: 310/940-1877 

Email: pines_michael@lacoe.edu 

Marcel Soriano 
Associate Professor 

Division of Administration and Counseling 

California State University, Los Angeles 

5151 State University Drive 

Los Angeles, C A 90032-8141 

Phone: 213/343-4381 

Fax: 213/343-4252 

Email: msorian@calstatela.edu 

Colorado 

James R. Craig 
Director 

Adamss Child and Family Services 
7840 Pecos St. 

Denver, CO 80221 
Phone: 303/853-3431 
Fax: 303/428-0233 

Texas 

Jenni Jennings 
Clinical Supervisor 
Youth & Families Centers 
Dallas Public Schools 
P.O. Box 4967 
Dallas, TX 75208 
Phone: 214/827-4343 
Fax: 214/827-4496 
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A PERSPECTIVE CN AGGCLNTING ECP PIEEEPENCES 
AND DISABILITIES IN ASSESSING STLPENTS 



The following are issues covered in our Center’s Introductory Packet on 
Assessing to Address Barriers to Learning (available at cost from the 
Center’s Clearinghouse). 



In gathering data related to intervention effectiveness, it is important to remember that some 
interventions are meant to change the school, classroom, home, and so forth. When interventions are 
designed to alter environments, then sound accountability focuses not just on assessing students but 
also evaluating environmental changes. Often, the primary need is to assess how well external 
barriers to learning have been addressed. 



When it is clear that student data are needed, the next consideration is whether the 
information already is in accessible, existing records 



If the desired information is not available and direct assessment of the student is 
appropriate, then concerns about the assessor's ability to gather valid information 
arise. When all is said and done, these concerns are reflected in three question: 

Are there valid procedures for gathering the information? 

(e.g., culturally appropriate instruments) 



Can the assessor establish a positive working relationship with the student? 



Relatedly, is the student motivated to provide the desired information? 



Concerns about cultural differences, disabilities, and other group differences 
resolve down to the problem of individual differences when it comes to assessing a 
given individual 
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ill. An Evaluation Flow Chart and Some 
Accompanying Material from a Resource 
Published by the National institute on Drug 
Abuse (Rockville, MD) entitled 



How Gccd is your Eras Abuse 
Treatment Eroeram? 

A Guide te Evaluation. 



Although the work stresses evaluation of drug treatment, the 
flow chart and related materials can be readily adapted to meet 
many program evaluation needs. 
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Outcome Study only 
Consult an Expert 




Exhibit III-2: Examples of Program Objectives 

Goal: Cygnus clients will develop a lifestyle that supports 

abstinence and enables them to provide healthy home environments 

for their children. 

Objectives: 

(1) Eighty percent of clients, before leaving treatment, will have 
completed a 6-week employment skills program that covers 
job search skills, interviewing skills, and basic work skills. 

(2) Eighty percent of clients with husbands or significant others, 
before leaving treatment, will have completed 12 weeks of 
couples counseling. 

(3) Eighty percent of clients will have completed a 6-week 
parenting ’skills program of 2 hourly meetings per week 
before leaving treatment. 

(4) Eighty percent of clients will have completed a 3 -part 
workshop on self-esteem within 45 days after entering 
treatment. 

(5) Eighty percent of clients will have completed a relapse prevention 
program within 6 months of completing primary treatment. 



Describing Your Drug Treatment Program 




34 



40 



Exhibit III-3: Eight Threats to Internal Validity 



1. HISTORY: Unplanned events that occur between the first and second measurements. 
Examples include changes in local drug use patterns, drug supply interruptions, changes 
in treatment approach, and seasonal drug use patterns. In general, the more time that 
elapses between measurements, the greater the threat from historical effects. 

2. MATURATION : Developmental changes that naturally occur in clients. Growing older, 
more experienced, or more independent may be especially important maturation effects 
with adolescent client populations. Other maturation effects raise particular concerns in 
samples from special populations in transition, such as pregnant women, HIV positive 
clients, and ex-offenders re-entering society from prison. 

3. TESTING: Effects of taking a measurement on the results of subsequent measurement. 
Repeated urine tests for drug use tend to discourage later drug use, for example, while 
repeated ability tests tend to raise scores as subjects practice their test-taking skills, even 
if no real increase in ability occurs. 

4. INSTRUMENTATION: Effects of changes in a measurement instrument, or in criteria 
for recording behavior, during the course of an evaluation. Common examples are a 
change in the cutoff point for a "drug positive” determination by a urinalysis laboratory 
or a change in police criteria for making arrests (for example, during a neighborhood 
crackdown). 

5. STATISTICAL REGRESSION: Effects of selecting samples on the basis of extreme 
behavior— over time, their behavior tends to "regress" toward the overall group average. 
In drug treatment program evaluation, prodrug attitudes in an extremely prodrug sample, 
heroin consumption in a sample selected during periods of heavy use, and self-esteem in 
a sample selected on the basis of low self-esteem, will all tend to be less extreme on a 
second measurement. 

6. SELECTION : Effects of unmeasured difficulties between a group receiving treatment and 
a nonequivalent group not receiving tr eatme n t Common examples of unmeasured 
difficulties include clients’ motivation to seek treatment, family and social support 
structures, and expectations about future drug use. The problem tends to arise when 
treatment is given to volunteers and withheld from nonvolunteers, instead of assigning 
volunteers randomly to treatment. 

7. ATTRITION: Effects of unequal dropout rates among different subgroups in the sample. 
In drug treatment program evaluation, common examples include differences between 
those who receive treatment and those who do not, between heavier drug users and 
lighter users, or between more and less satisfied clients. 

8. HAWTHORNE EFFECT: Effect of changes which are due to the fact of being included 
in an evaluation. 




Choosing an Evaluation Daaign 
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IV. Two Reports from the Center for School Mental 
Health Assistance (Prepared by Mark Weist, Ph.D.) 



Critical Issue Planning Session: 

>■ Quality Assurance 

> Documenting Effectiveness of School Mental 
Health Programs 
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Center for School Mental Health Assistance 
Critical Issue Planning Session: Quality Assurance 
March 26, 1996 
Baltimore, Maryland 

Meeting Overview 

One of the seven major objectives of the Center for School Mental Health Assistance (CSMHA) 
!^m° V,de dlrec r IIOn in addf essing issues critical to the advancement of school mental health 
(SMH; services. In year I of the project, three issues have been targeted as the most important: 
1) UeveI oping mechanisms to assure the quality of SMH services, 2) documenting the 
effecti veness of SMH services, and 3) developing stable funding mechanisms. To begin to 
formulate strategies to address these pressing issues, the CSMHA will convene analysis and 
planning sessions with national experts. The first in our series of these meetings focused on 
Quality Assurance, and was held on March 26, 1996. The meeting included national experts 
representing providers and planners of school health and mental health services (Paula 
Armbruster, MSW; Lois Flaherty, MD; Ellen Garrison, Ph.D.; Marcia Glass-Siegel. LCSW-C; 
Bernice R °senthal ? M.P.H.; Beth Warner, Ph.D., Mark Weist, Ph.D.), leading academicians (Stan 
ne roan, • • •» 1 Leaf, Ph.D), representatives from local and state agencies and governments 

(An Cohen, J.D.; Bonnie Peet, MSN; Tom Merrick, M.A.), leaders in managed care 
organizations (Kirk Griffith, Ph.D., Donna Horsey, R.N.), family members of youth with 
emotional and behavioral disorders (Susan Tager, B.A.), quality assurance officers (Eileen 
Hastings, LCSW-C; Patrick Myers, M.S., Julie Tyler, R.N.) and others (Patrice Davis-Rose). 

L Vision for School Mental Health Services 

Our initial discussion focused on the Vision for SMH services. Important points that were raised 
during this discussion included: 

Major dimensons of Quality Assurance (QA) are Access and Availability, the Quality of 
Interventions. Continuity of Care, Collaboration between Providers, Acceptability of 
Services to Consumers, and Reasonable Cost. Participants endorsed these dimensions as 
being relevant to SMH programs. 

To ensure quality, SMH services should be integrated with community health services, 

and reflect genuinely collaborative relationships between mental health, health, and 
educational staff. 



There is an important distinction between Quality Assurance (QA) and Quality 
Improvement (QI). QI j s a gradual and progressive improvement in the process of care 
toward meeting long-term objectives, while QA is more of a snapshot of how an 
organization is functioning at a given point in time. QI never ends; instead, process is, 
in uenced by objectives, and objectives are influenced by the process in a continuing 
cycle of service improvement. 
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A commitment to quality requires linking efforts to customers, but identifying them is a 
challenge. Who are the customers: students, their families, legislators? The vision will 
vary based on the customers. 

Any system of QI for SMH services must be centered around the significant 
improvements in access enabled by having them in schools. There are numerous barriers 
preventing youth from receiving services in traditional settings (e.g., community mental 
health centers, private offices). As recognition of these barriers has increased, so have 
activities to develop more and more primary mental health services for youth in schools. 
As such, QI programs should not impose barriers between youth and needed services. 

Efforts to document the effectiveness of SMH programs are actually subsumed under QI. 
Often education staff and community members need to be educated on the benefits of 
school-based mental health services (e.g., to address aggressive behavior and violence). 
The benefits of SMH services must be connected to outcomes pertaining to educational 
functioning. Efforts to document the cost savings and general community benefits (e.g., 
in assisting the juvenile justice system) of these programs are also critically needed. 

The degree to which SMH programs address pressing problems in their surrounding 
communities (e.g., teenage pregnancy, violence) is an important dimension for measuring 
their quality. There is no way around significant community input that comes from 
parents and leaders from community clubs and religious groups. 

Ideally, SMH programs should emphasize primary and secondary prevention of 
psychosocial adjustment problems in youth. But in many cases, there will be a need for 
tertiary preventive services, and the needs of youth with severe/chronic disturbances 
cannot be neglected. As such, SMH programs should not replicate other services in 
communities, but be meshed with them. 

To truly have an impact, SMH programs need to be accepted by school staff and the 
teachers union. One method to promote such acceptance is to highlight the benefits of 
SMH programs in "enabling" children’s learning (from H. Adelman and L. Taylor). This 
connection is absolutely critical to political advocacy efforts for SMH services. 

Families need to be closely involved in all phases and aspects of planning for and 

providing SMH programs. Outreach efforts to families, while often frustrating, are 
critically important. 



II. Small Group Discussions: Translating the Vision of Quality SMH Sendees into 
Operational Principles and Procedures 

In the second half of our meeting we broke into discussion groups which were charged with 
^'nationalizing the vision for high quality SMH sendees in the major domains of: A Goals. B. 
ERJpe of Sen ices, and C. Quality Indices. These are reviewed here: 
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A. Goals for SMH Services: 



where there clear problems with access to mental health 
services tor youth (e.g., very rural areas, inner cities), 

™: em0t ‘ 0naJ ’ behavioral ™ d educational functioning for youth 

provide high quality yet cost-effective services, 

Msme P ATtt^kes V ' de SCh ° 01 h !f' th f d mentai heaith se ™ ces “ en integrated fashion. 
" , “ at semces integrated with services available in the community 

sensitive, 3 ' aSS “ Sment “ d trea “ nenl services “« b °d> developmentaily and’ culturally 
work to reduce the stigma associated with mental heaith services 

^o'cme fn U ‘ mt ° 46 re0r8ani2a ‘ ion “ d imptovement of school "support" services 
toppeTouf 3PProPnate Se " iC “ ^ Wh ° " "** 0Ut “ d foryouth w"o have 

ISo the rT 131 “r* Pr08ram in ‘° *' P hi,0S0 P h >' »d life of the school, 
nroviH, h relevance °f treatment targets and provide empirically valid services 

health Iraff Sm3 “ C effeC ‘ iVe aSSis ‘ ance “ educa,ion - "dated mentai health, and school 
Scope of SMH Services: 

conduct local needs assessments to ensure that services are congment with community 

preventive services, based on systems of early identification of youth in need. 

ion to more preventive services, identify youth with mild as well as severe 
emotionai/behaviorai disturbances, as well as sev ere 

provide services regardless of ability to pay, 

«Zm ^L c °eTtTd? LTir indu h dins assessmem md services ,o yo«* 

“ d treatmem services t0 address substance-related issues, including 
g and alcohol abuse, exposure to familial substance abuse, and drug dealing * 

trsssa » 

^enMMdT”^ ° Utr “ Ch activities workin 8 with neighborhood associations 
p nt-teacher organizations) to promote community-wide benefits of SMH services ’ 

ZZT Pmem ° f " SCh ° 0, - Wide " Sen ' iCeS ,0 " ddreSS P-^ P^Lms ie g.. 
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* provide mental health education and/or facilitate the development of mental health 
programs in educational curricula. 

develop mental health screening to facilitate mental health services for youth who use 
school health services at high rates, 

develop mental health programs for youth contending with chronic illness and/or special 
health problems. 

C. Quality Indices: 

* amount of time between referral of students and contact with clinicians. 

* appointment keeping rate of referred students, 

* percentage of referred students who are actually seen by the clinician, 

* number of students who are successfully referred to appropriate services in the 
community (e.g., for medication), 

amount of services provided (e.g., number of students seen; number of individual, group 
and family therapy sessions), 

degree of family involvement (e.g., percentage of students seen with contact between the 
clinician and the family), 

* educational and emotional/behavioral improvement in youth from pre to post 
intervention, 

"consumer satisfaction" ratings from youth, families and educational staff, 

* number or percentage of youth referred for special education services, 

number or percentage of youth who undergo changes in intensity of special education 
services (e.g., from higher to lower), 

* having an orientation program for new clinicians, 

"sensitivity" of mental health providers to factors important to the school setting, 
functioning of multidisciplinary teams (e.g., disciplines represented, yearly objectives 
accomplished), 

degree to which mental health staff become accepted by, and part of, the school milieu, 
amount of appropriate (e.g., by licensed providers) clinical supervision of the school- 
based therapist, 

* expertise and experience of school-based clinicians, 

support provided to the SMH program by the school administration (e.g., adequacy of 
office space), 

* compliance of the program with relevant state laws and professional guidelines. 

III. Summary Discussion 



In summing up the day, four "overarching goals" for SMH programs were identified: 1) Provide 
comprehensive direct clinical assessment and treatment services for underserved youth, 

2) emphasize preventive programs that provide early identification and treatment for vouth in 
need. 3) ensure that mental health programs have a strength or competency focus, versus an 
exclusive focus on reducing psychopathology, and 4) seek to maximize the impact of mental 
health services bv involvement in collaborative efforts aimed at improving the global school 
O ironment. 

EMC . : 40 4 o' 
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In a similar process to the above, critical SMH services were identified. At the core, SMH 

should include focused mental health evaluation, which acknowledges the influence of 

services bwi! ‘"‘i!^, c,a jf r00m beh avioral observations), and leads to referral for appropriate 
' ' * e sch ° o1 and/or community when indicated. Mechanisms should also be in place to 

track referrals to ensure that youth in fact contact needed services. At the next level of 

sTc'STnH arC: f tr “ tmem servic “’ “eluding individual group and family therapy for youth in 

to hroadc ^ 6 u< '? t ’ on * b) consultative and mental health education services designed 

would com* 6 ‘T“l 0fl th .% SMH program - There was § enera l consensus that these elements 
would comprise the "basic” SMH program. 



m.weist 

4/15/96 

(revised 5/5/96) 
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Center for School Mental Health Assistance 
Critical Issue Planning Session II 

"Documenting the Effectiveness of School Mental Health Programs" 

June 18 & July 23 1996 
Baltimore, Maryland 

Meeting Overview 

On June 18, 1996, the Center for School Mental Health Assistance (CSMHA) convened a meeting of national experts 
in Baltimore, Maryland to explore relevant issues and methods to document the effectiveness of school mental health 
(SMH) services. This meeting was the second in a series of meetings designed to analyze and address critical issues 
confronting the movement to develop and improve SMH services nationwide, and built on concepts developed in our 
first meeting in March, 1996 focusing on Quality Assurance. Participants in the meeting included providers and 
planners of school mental health services (Paula Armbruster, MSW; Ellen Garrison, PhD; Marcia Glass-Siegel, LCSW- 
C; Bernice Rosenthal, MPH; Mark Weist, PhD), education administrator (Louise Fink, MEd), experts on mental health 
evaluation (Louis Hagopian. PhD; Pat Myers, MA; Jennifer Oppenheim, PhD) and children's mental health issues 
(Cheryl Alexander. PhD; Joan Dodge, PhD; Marsha Gorth, LCSW-C; Eileen Hastings, LCSW-C; Phil Leaf, PhD), and 
family member (Gail Johnson). 

I. Open Discussion 

To open the meeting, participants shared their experiences with, and thoughts about SMH evaluation. Important points 
raised during this discussion included: 

• One benefit of managed care may be to increase pressure on SMH programs to develop systems to evaluate 
provided services. Managed care companies are looking for proof that the dollars they have invested are 
actually paying for something. Evaluations showing that SMH programs are effective should enhance the 
willingness of more managed care companies to fund them. 

• University-based programs may have reputations of "using kids as research subjects," which may lead to 
resistance by schools in cooperating with evaluation plans. 

• There are numerous potential obstacles to implementing formal evaluation designs. These include obtaining 
parental approval and consent for the project, finding and assessing a comparison group, addressing language 
barriers for certain students (e.g., Hispanic youth), and dealing with poor reading levels of many students. 

• Need careful assessment of the importance of the particular outcome to be assessed, respondent burden to 
obtain measures of this outcome, and pragmatic factors in obtaining the outcome measure (e.g.. Is it brief?, 
Will using it lead to meaningful findings that help.improve the program?). 

• Many programs initiate complex and labor intensive evaluation designs, collect voluminous amounts of data, 
then actually only use a small percentage of the data. This essentially leads to wasted efforts, which should 
be avoided at all costs. 

' The concept of Treatment Utility is an important one which states that the evaluation process should directly 

contribute to positive outcomes, and definitely should not lead to negative outcomes (e.g., students dropping 
out due to very long assessment procedures). 

1 Evaluation plans need to be tailored to the individual school, as "every school has its own ecology, and every 

Board of Education has its own ecology.” 

1 Community members and parents should be closely involved in the development of SMH evaluation plans. 
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* Should not view these evaluation plans as research; rather, they should be viewed as part of clinical assessment, 
which will assist in improving the quality of care. Two major functions of evaluation are to design effective 
programs for kids, and to document that services are beneficial for support of the program. 

* For every evaluation program, a careful pre-analysis should occur which considers the goals of the evaluation 
and the means to accomplish the goals. As few measures as possible should be used to assess accomplishment 
toward goals. 

U. Important Factors to Consider in Developing SMH Evaluations 

The panel generated a list of principles that characterize good SMH evaluation programs. These principles included: 

* Being relevant to the type of services provided and the population served, 

* having an evaluation process that would be generalizable to different programs and different student 
populations, 

* viewing evaluation as an ongoing process, which provides feedback to efforts to continuously improve 
services, 

* attending to cultural sensitivity in evaluation processes and measures, 

* involving key "stakeholders" in the evaluation process, such as students, families, school staff, and funders, 

* including multiple levels of assessment, for example, measuring student grades and absenteeism, as well 
satisfaction of teachers with the program, 

* being relatively simple and "doable," 

* focusing on factors that are likely to be affected by the program, and 

* using measures that are "face valid,” or make sense to those completing them. 

III. Student-Focused Evaluation Programs 

The panel discussed student characteristics and indices of functioning that are most relevant to SMH programs. These 

included: 

* Commonly occurring "emotional” problems such as depression, anxiety, traumatization symptoms, and social 
withdrawal, 

* commonly occurring behavioral problems such as aggression, oppositionality, classroom disruptiveness, and 
hyperactivity, 

* school performance in terms of grades, scores on standardized tests, attendance, and discipline problems, 

* family and peer relationships, and 

* competencies or qualities of "resilience" such as social skills, positive self-concept, involvement in meaningful 
activities, participation in athletics, spirituality, coping skills and problem-solving. 
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There was some discussion of the fact that frequently in mental health, there is a bias toward focusing on pathology, 

and that positive qualities of youth are often neglected in evaluation programs. There was strong support for a focus 

on competency variables in SMH program evaluations. 

IV. Ethics of SMH Evaluations 

Over lunch, participants broke into small groups to discuss ethical issues involved in evaluations of SMH programs. 

Reports of these discussion groups emphasized the following points: 

• Parental consent for services is essential. 

• Family members should be involved in the development of SMH services, and should provide feedback on 
them once they are developed. 

• There is an ethical requirement to provide treatment when screening programs are initiated. As such, programs 
should be cautious about implementing broad screening projects if they have limited resources to provide 
follow-up services. 

• Special safeguards are necessary to protect the confidentiality of students receiving SMH services (e.g., the 
fact that they are receiving mental health services can usually be witnessed by other students). 

• Programs need to document that youth are receiving enough from a program to justify an intensive evaluation. 

• When professional staff are not involved in the collection of data, protections need to be in place to ensure its 
confidentiality. 

• Whenever possible, programs should use reliable and valid measurement tools, versus created ones, to avoid 
the possibility of anomalous findings. 

V. Caveats to be Aware of When Implementing SMH Evaluations 

The panel reviewed pitfalls of evaluation, and problems that are encountered in evaluation efforts. These included: 

• The psychotherapy outcome literature has failed to document that interventions as commonly implemented in 
applied settings are actually effective. When an SMH evaluation yields negative results, this information can 
and should be used to improve the program. However, there is a danger that if such negative results are 
disseminated, they could be used to justify a cut in funding. 

• Many youth show up for mental health appointments in schools during crises. Overtime, these youth would 
do better without intervention. As such, we need to be careful to ascribe recovery from a crisis to SMH 
services. 

’ In some SMH programs, clinicians are so overwhelmed with students in need, that less than optimal services 

are able to be delivered to any one student This dilutes the effect of school-based interventions, and may lead 
to failure of evaluations to document positive changes. 

' A common failure in evaluation programs is the failure to assess the integrity of treatment services. For 

example, in many programs little is known about the skill level of the therapist, and what happens behind the 
therapy door. In structured programs to address particular student problems (e.g., anger control), clinicians 
have been shown to deviate from the prescribed treatment program without structure (e.g., treatment manuals) 
and ongoing monitoring of their performance. 
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* Asa general rule, SMH programs are not fine-tuned interventions, which decreases the likelihood of finding 
positive impacts. 

* Many of the instruments used in child mental health assessment basically provide information on whether the 
child does or does not meet criteria for a specific problem (i.e., "caseness"). These measures are commonly 
not sensitive to treatment effects. Similarly, there is a lot of random movement from "case" to "noncase" and 
caution is needed in ascribing this movement to an SMH program. 

The above points highlight complexities and difficult issues involved in SMH evaluation, particularly evaluations 
focused on treatment outcomes. As such, programs should be cautious in attempting to document treatment impacts. 
However, assessment of treatment outcomes is only one aspect of program evaluation. Such broader program evaluation 
was the next focus of group discussion. 

VI. Desired Outcomes for SMH Programs for Relevant Stakeholders 

A. In this section of the meeting, participants sought to address the question: What needs to be demonstrated for SMH 
programs to expand? The group identified major stakeholders for SMH programs of: I) Children and families, 2) 
schools, 3) community health and mental health systems, and, 4) funders. Desired outcomes for each of these 
stakeholders are reviewed in the following: 

L Children and Families 

The group (which included family members of children with emotional/behavioral disturbances) suggested the following 
outcomes as being the most important. From involvement in an SMH program, children should show: 

* academic success 

* enhanced self-esteem 

• improved social skills 

• improved capacity to function independently 

* more positive behaviors at home and school 

• decreased levels of emotional disturbance 
2. Schools 

Participants (including education administrators) suggested that schools are most interested in these outcomes from SMH 
programs: 

academic success 

improved school attendance 

reduced school violence and aggression 

established linkages between the child and other needed services 
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3. Community Health/ M ental Health System : 

Administrators and planners of community health and mental health services were viewed to prioritize the following 
outcomes for SMH programs: 

• detection and treatment of emotional/behavioral problems early 

• prevention of emotional and behavioral problems 

• family preservation 

* decreased child abuse 

* decreased suicidal behavior in youth 

* decreased substance abuse by youth 

• fewer entries into the juvenile justice system 
4. Funders 

The group suggested that funders (e.g.,. third party payers, managed care companies) would have one major outcome 
desired for SMH programs: 

* reduction of high cost services such as hospitalization 

B. The panel then attempted to identify the most desired outcomes for SMH programs; that is, outcomes that cut across 
the various stakeholder groups. Five outcomes were determined to be the most important. These were: 

1 . Prevention of emotional/behavioral problems and early intervention to prevent their worsening, 

2. improved school attendance, 

3. decreased risk-taking behavior including substance abuse and violence by youth, 

4. decreased use of high intensity services, including community health services such as hospitalization, and school 
services such as placement in special education, and 

5. improved collaborative linkages for programs within schools, and between school-based and community-based 
programs. 

In discussion of these outcome indicators, the panel emphasized that reduction of emotionalbehavioral symptoms in 
youth per se, is not a priority outcome; rather, positive changes in these symptoms should impact youth functioning in 
other domains as above. Thus, symptom reduction should be viewed as a means to an end, versus and end in and of 
itself. 



VII. Meeting Day Two - Open Forum 

CSMHA staff presented highlights of the Day One meeting, and invited general comments from the panel, which 
included ail but four participants from the first meeting (Paula Armbruster, Marsha Gorth, Louis Hagopian, and Jennifer 
Oppenheim). Noteworthy comments from the panel included: 



O 

ERIC 



46 



52 



BEST COPY AVAILABLE 



Page 6 



• The design and conduct of treatment outcome studies are highly complex and fraught with problems. These 
problems include the general failure of the few existing studies to document significant benefits, the 
considerable variability that occurs in service delivery not only across, but within programs, and the generally 
limited resources of SMH programs to mount these kinds of studies. In essence, most SMH programs should 
be interested in evaluating the effectiveness of programs, not in evaluating the efficacy of particular treatments . 

• In designing SMH programs, there is a need to analyze how the school-based program will affect the 
community. For example, a program could reduce school violence by expelling violent students, but this 
would likely cause problems in the community. In SMH evaluations, there needs to be attention to positive 
and potentially negative community impacts. 

• If SMH programs strive to target "resilience factors," or variables that have been shown in the literature to 
promote positive psychosocial adjustment in youth under stress (e.g., family support, social skills, involvement 
in meaningful activities), the likelihood that these programs will show positive and meaningful impacts should 
be improved. 

• There is a continuum of SMH programs in terms of their sophistication and resources available to them. 
"Cadillac" programs will have a much greater chance of documenting program benefits than smaller, more 
isolated programs. In essence, program planners should consider the "minimum threshold for evaluation” to 
avoid the worst case scenario of evaluating a small program, finding negative results, and then losing funding. 
Generally, the group recommended that comprehensive evaluations only be conducted for programs (i.e., not 
one part-time clinician in one school), that preferably have some institutional backing (e.g., from a university 
or community health/mental health agency). 

• Evaluation should be tailored to the size and nature of the SMH program. Small programs should emphasize 
evaluation of the impacts for individual children, and should not undertake systems evaluations, as systems 
level changes will probably not be shown. Systems level evaluations should be limited to larger programs with 
more resources. Essentially, SMH programs should conduct a "self-evaluation process," to guide decisions 
about the appropriate evaluation strategy. 

• One problem that has severely constrained SMH program evaluation has been the lack of funding to do this. 
Among participants, no one was aware of any SMH program receiving funding for evaluation. Clearly, such 
funding is needed to document program strengths and weaknesses. Lobbying efforts will be important to sell 
funders on the benefits of, and significant need for SMH program evaluation. 

VIII. A Recommended Process for SMH Program Evaluation 

A. CSMHA staff presented a proposal to the panel on a process for SMH program evaluation. The panel endorsed and 

fine-tuned this process, which follows: 

1. Define the program (e.g., number of clinicians, funding, provided services). 

2. Define stakeholders for the program and determ in their interests and goals. 

3. Develop program goals so that they reflect interests of stakeholders. 

4. Develop a realistic evaluation plan, focusing on outcomes that are of interest to stakeholders, and that can be 

collected within the pragmatic constraints of the program. 

5. Gain feedback from the stakeholders on the evaluation plan and modify the plan based on this feedback. 

6. Implement the evaluation plan and monitor its implementation. 
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7. Organize program evaluation findings. 

8. Present program evaluation findings to representative stakeholders for their feedback and input. 

9. Modify and improve the program based on results of the evaluation. 

B. To elaborate on such a plan with concrete examples, a subgroup of the panel agreed to develop an SMH program 
evaluation based on the Baltimore City experience. The goal is for this project to commence in the Fall of 1996, and 
to be finished in the Summer of 1997. This prototype SMH evaluation project will then be disseminated nationally in 
Year 3 of the CSMHA grant (beginning 10/1/97). 

IX. Other Steps 

In addition to the pilot SMH program evaluation project reviewed above, the CSMHA will be gathering and organizing 
resources on this topic. A number of participants from this meeting will as sist in this process, as well as recruiting other 
knowledgeable people for assistance. The panel endorsed the notion of a Center developed manual for SMH program 
evaluation. 

m.weist 
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We hope you found this to be a useful resource . 

There's more where this came from! 

This packet has been specially prepared by our Clearinghouse. Other Introductory 
Packets and materials are available. Resources in the Clearinghouse are organized around 
the following categories. 

CLEARINGHOUSE CATEGORIES 



Systemic Concerns 



• Policy issues related to mental health in schools 

• Mechanisms and procedures for 

program/service coordination 

• Collaborative Teams 

• School-community service linkages 

• Cross disciplinary training and 

interprofessional education 

• Comprehensive, integrated programmatic 

approaches (as contrasted with fragmented, 
categorical, specialist oriented services) 

• Other System Topics: 



• Issues related to working in rural, urban, 

and suburban areas 

• Restructuring school support service 

• Systemic change strategies 

• Involving stakeholders in decisions 

• Staffing patterns 

• Financing 

• Evaluation, Quality Assurance 

• Legal Issues 

• Professional standards 



Programs and Process Concerns: 



• Clustering activities into a cohesive, 

programmatic approach 

• Support for transitions 

• Mental health education to enhance 

healthy development & prevent problems 

• Parent/home involvement 

• Enhancing classrooms to reduce referrals 

(including prereferral interventions) 

• Use of volunteers/trainees 

• Outreach to community 

• Crisis response 

• Crisis and violence prevention 

(including safe schools) 

• Other program and process concerns: 



• Staff capacity building & support 

• Cultural competence 

• Minimizing burnout 

• Interventions for student and 

family assistance 

• Screening/ Assessment 

• Enhancing triage & ref. processes 

• Least Intervention Needed 

• Short-term student counseling 

• Family counseling and support 

• Case monitoring/management 

• Confidentiality 

• Record keeping and reporting 

• School-based Clinics 



Psychosocial Problems 



• Drug/alcoh. abuse 

• Depression/suicide 

• Grief 

• Dropout prevention 

• Learning Problems 
•School Adjustment (including 



• Pregnancy prevention/support 

• Eating problems (anorexia, bulim.) 

• Physical/Sexual Abuse 

• Neglect 

• Gangs 

newcomer acculturation) 



• Self-esteem 

• Relationship problems 

• Anxiety 

• Disabilities 

• Gender and sexuality 

• Reactions to chronic illness 



• Other Psychosocial problems: 
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